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Automation  and  Cognition  in  Air  Traffic  Control; 
An  Empirical  Investigation 


In  recent  years,  the  17,000  civilian  air  traffic  con¬ 
trollers  in  the  United  States  have  handled  upward  of 
100  million  aircraft  operations  annually.  Further  sig¬ 
nificant  increases  in  traffic  volume  are  expected  over 
the  next  few  years,  which  will  place  additional  de¬ 
mands  on  the  available  airspace  and  will  exact  higher 
performance  from  air  traffic  controllers.  To  meet  these 
traffic  loads,  far-reaching  automation  is  scheduled  for 
introduction  into  air  traffic  control  (ATC)  during  the 
remainder  of  the  1990s.  There  is  little  doubt  that 
automation  is  inevitable  and,  in  the  end,  largely  ben¬ 
eficial.  Nonetheless,  numerous  potentially  important 
concerns  have  been  expressed  about  the  psychological 
consequences  of  automation  on  air  traffic  controllers. 
These  concerns  typically  revolve  around  the  view  that 
the  manual  performance  of  some  tasks,  although  re¬ 
petitive  and  burdensome,  may  be  beneficial  because  it 
builds  understanding  relevant  to  overall  performance. 
In  this  view,  automation  of  routine  activities  may  have 
unintended  negative  consequences  for  performance 
and,  by  implication,  air  traffic  safety  (e.g.,  Hopkin, 
1988,  1989). 

We  have  focused  on  the  likely  psychological  conse¬ 
quences  associated  with  the  initial  stage  of  automation 
of  “en  route”  ATC  (see  Vortac  &  Gettys,  1991,  for  a 
review).  Briefly,  en  route  control  handles  the  airspace 
between  departure  and  arrival  airports  and  corresponds 
approximately  to  the  high-speed  and  high-altitude 
cruise  between  takeoff  and  landing.'  En  route  control 
is  distributed  across  some  20  ATC  centers  in  the 
continental  United  States,  each  of  which  is  responsible 
for  a  large  segment  of  airspace.  The  airspace  assigned 
to  an  en  route  center,  in  turn,  is  carved  into  multiple 
sectors,  each  handled  by  a  controller  or,  if  required  by 
traffic  loads,  by  a  team  of  two  or  more  controllers. 


Currently,  en  route  control  relies  on  three  primary 
tools:  Computer-enhanced  radar  information  displayed 
on  the  plan  view  display  (PVD),  communication  de¬ 
vices  (radio  and  telephone),  and  flight  progress  strips 
(FPSs).  FPSs  consist  of  small  rectangular  pieces  of 
paper  containing  up  to  31  pieces  of  flight  data  (e.g., 
call  sign,  aircraft  type,  assigned  altitude,  route  of 
fight,  etc.)  that  augment  the  information  obtainable 
from  the  PVD.  (See  Figure  1.) 

In  the  field,  each  flight  is  typically  represented  by  a 
unique  FPS,  printed  prior  to  entry  into  the  sector  and 
based  primarily  on  the  flight  plan  filed  before  take-off. 
Strips  are  mounted  in  plastic  holders  that  are  stacked, 
usually  by  arrival  times,  in  ascending  chronological 
order  in  a  posting  board  or  “bay”  located  next  to  the 
radar  display.  Flights  about  to  enter  a  sector  are  posted 
in  a  suspense  bay;  once  a  flight  becomes  active  by 
entering  the  controller’s  sector,  its  strip  is  moved  to  the 
adjacent  active  bay.  While  a  flight  is  active,  the  con¬ 
troller  frequently  interacts  with  the  corresponding 
FPS  on  the  board  by  writing  on  the  strip  itself  to 
update  the  speed,  the  altitude,  the  route,  or  by  moving 
the  FPS  within  the  active  bay.  Figure  1  shows  the 
difference  between  a  strip  immediately  before  a  plane 
enters  the  controller’s  sector  and  that  same  heavily 
marked  strip  after  the  controller  has  taken  a  number  of 
actions  involving  that  aircraft.  Some  of  these  markings 
are  due  to  the  controller’s  legal  obligation  to  use  the 
strip  as  a  legal  record  of  the  flight.  Thus,  while  control¬ 
lers  interact  with  the  paper  FPSs  at  least  part  of  this 
interaction  is  because  it  is  mandated.  The  question 
remains  whether  this  mandated  activity  has  coinciden¬ 
tal  cognitive  and  performance  benefits. 


'  Different  phases  of  a  typical  flight  are  handled  by  a  number  of  different  air  traffic  control  facilities.  Ground  control  separates  aircraft 
as  they  taxi  from  the  gate  to  the  active  runway.  Local  control  grants  permission  to  enter  an  active  runway,  anH  then  handles  ’nkeoff  and 
initial  climb-out  to  (typically)  3,000  feel.  From  this  point  out  to  within  50  nautical  miles  of  the  airport  is  the  domain  of  the  Terminal  Radar 
Approach  Control  fTRACON).  The  remaining  airspace  is  the  domain  of  en  route  control,  which  includes  arrivals  and  departures  from 
uncontrolled  airports. 
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Figure  1.  A  flight  progress  strip  for  a  flight,  before  it  enters  the  controller’s  sector  (top)  and  after  it  has 
been  active  in  the  controller’s  sector  (bottom). 


The  first  stage  of  en  route  automation,  the  Initial 
Sector  Suite  System  (ISSS),  has  its  most  dramatic 
effect  on  the  FPS.  Under  ISSS,  FPSs  will  be  replaced  by 
entries  on  a  large  electronic  display  located  next  to  an 
improved,  colorized  version  of  the  PVD.  Controllers 
will  interact  with  flight  data  in  an  indirect  manner 
through  a  keyboard,  rather  than  by  direct  physical 
manipulation  and  updating  of  an  FPS.  Moreover, 
numerous  current  strip-related  activities  (e.g.,  moving 
strips  between  bays)  will  be  obviated  by  electronic 
processes  (e.g.,  automatic  posting  of  strips).  Finally, 
less  information  is  expected  to  be  shown  on  the  elec¬ 
tronic  displays  than  is  currently  visible  on  paper  strips. 

Concerns  have  been  voiced  about  the  impending 
removal  of  paper  flight  strips,  based  on  the  possibility 
that  some  of  their  unique  and  potentially  beneficial 
properties  might  not  be  fully  embodied  in  the  new 
electronic  system.  In  particular,  it  has  been  claimed 
that  the  strips  may  embody  important  communicative 
purposes  (Shapiro,  Hughes,  Randall,  &  Harper,  1991), 
that  they  serve  as  a  memory  aid  (e.g.,  Hopkin,  1989, 
1991;  Jackson,  1989),  that  they  are  important  external 
retrieval  aids,  and  that  they  support  cognitive  pro¬ 
cesses  in  numerous  additional  ways  (for  a  review  see 


Vortac  &  Gettys,  1991).  It  thus  stands  to  reason  that 
the  conversion  from  paper  FPSs  to  electronic  displays 
may  have  adverse  cognitive  consequences.  However, 
like  other  assertions  surrounding  automation,  the  ex¬ 
act  role  of  the  strips  has  not  been  resolved  by  empirical 
testing. 

One  reason  for  this  paucity  of  research  may  be  the 
difficulty  in  determining  the  exact  final  functionality 
of  the  automation  that  will  replace  the  paper  FPSs 
(Nordall,  1993).  Although  the  general  framework  for 
the  new  system  has  long  been  determined  (Ammerman 
&  Jones,  1988),  considerable  debate  continues  to 
revolve  around  the  optimal  format  in  ISSS  for  the  new 
electronic  “strips.”  To  accommodate  this  design  un¬ 
certainty  without  compromising  a  meaningful  test  of 
the  likely  consequences  of  ISSS,  the  experimental 
manipulation  reported  here  was  chosen  to  be  more 
extreme  in  its  potential  cognitive  consequences  than 
the  anticipated  automation. 

We  observed  controllers  under  one  of  two  condi¬ 
tions:  Subjects  in  the  automation-analog  (Restricted) 
condition  were  given  limited  information  on  their 
strips,  and  they  could  not  move  or  write  on  them.  The 
control  (Normal)  subjects  had  full  access  to  unmodi- 


2 


Autemation  mmJ  Cognitivn 


fled  strips.  Hence,  the  results  of  the  present  experi¬ 
ment  should  be  interpreted  as  an  upper  bound  on  the 
cognitive  changes  attributable  to  interaction  with  FPSs. 

In  addition  to  measuring  subjects’  air  traffic  control 
performance,  the  experiment  also  involved  a  variety  of 
cognitive  measures.  Cognitive  measures  are  often  more 
sensitive  than  conventional  performance  measures; 
this  is  particularly  true  in  ATC  where  serious  proce¬ 
dural  errors  and  other  overt  performance  deficits  are 
extremely  rare.  Hence,  well-chosen  cognitive  measures 
may  serve  to  identify  deficiencies  before  they  result  in 
rare — but  nonetheless  catastrophic — failures. 

We  now  identify  cognitive  processes  likely  to  be 
affected  by  the  automation  of  flight  strips,  thus  mea¬ 
sured  in  our  experiment.  A  comprehensive  review  of 
the  role  of  cognitive  processes  in  air  traffic  control, 
going  beyond  the  role  of  FPSs,  is  provided  by  Cox 
(1992). 

Attention 

Attentional  engagement.  An  important  part  of  assess¬ 
ing  the  impact  from  a  cognitive  perspective  of  any 
automation  is  to  determine  if  the  automation  affects 
the  extent  to  which  the  controller  is  attentionally 
engaged  in  the  primary  task  of  controlling  air  traffic.  In 
psychology,  momentary  processing  capacity  is  com¬ 
monly  measured  by  a  secondary  probe  task  in  which 
latency  to  respond  to  some  secondary  (i.e.,  less  impor¬ 
tant)  stimulus  is  thought  to  reflect  the  amount  of 
attentional  capacity  devoted  to  the  primary  task 
(Baddeley  &  Hitch,  1974).  Estimates  of  capacity  de¬ 
mand  for  any  particular  task  depend  on  particular 
characteristics  of  the  probe  task  (McLeod,  1977, 1978) 
and  thus  the  latency  to  respond  reflects  the  cost  of 
attentional  </»engagement  from  the  primary  task 
(Allport,  1989).  Response  time  to  the  probe  is  slow 
when  a  high  level  of  selective  attention  is  necessary  to 
maintain  the  integrity  of  the  primary  task. 

Thus  it  is  important  not  to  view  the  secondary 
probe  as  providing  a  passive  indirect  measure  of  pro¬ 
cessing  capacity  of  the  primary  task.  To  assess  engage¬ 
ment  from  such  a  task,  it  is  important  that  the  probe 
be  part  of  the  high  fidelity  simulation.  Fortunately,  in 
ATC,  responding  to  telephone  calls  from  adjacent 
centers  is  a  routine  component  of  controlling  traffic. 


giving  us  a  naturalistic  secondary  probe  and  an 
uncorrupted  measure  of  the  cost  of  attentional  engage¬ 
ment. 

Visual  search.  Automation  of  FPSs  is  likely  to  influ¬ 
ence  visual  search  rate.  Visual  search  involves  alloca¬ 
tion  of  limited  attentional  resources,  in  order  to  select 
from  a  number  of  potential  inputs,  the  stimulus  to  be 
attended  (Johnston  &  Dark,  1986;  Shiffrin,  1988). 
When  incoming  information  can  be  processed  with 
little  attentional  processing,  it  appears  to  “pop-out” 
from  the  background  (Treisman  &  Gelade,  1980).  An 
FPS  that  has  been  marked  extensively  is  visually  dis¬ 
tinctive  and  may  pop  out  from  among  the  other  strips 
(Figure  1 ).  For  example,  if  assigned  altitude  is  changed, 
the  controller  crosses  out  the  original  value  and  writes 
in  the  new  altitude  underneath.  Often,  a  flight  will 
require  a  number  of  such  changes;  their  cumulative 
effect  being  a  visually  unique  strip  that  can  be  more 
easily  located  in  the  strip  bay  (Jackson,  1989).  Under 
automation,  however,  all  the  electronic  strips  will  look 
much  the  same;  the  history  will  not  be  maintained  (at 
this  writing,  under  the  operative  design  for  ISSS,  flight 
history  appears  to  be  limited  to  the  preservation  of  the 
last  preceding  entry,  and  the  computer  forces  stan¬ 
dardization  in  appearance.)  This  may  slow  search  of 
the  strip  bay  under  automation,  where  an  automatic 
“pop-out”  of  the  visually  distinctive  information  might 
have  occurred  previously. 

Search  may  also  be  aided  because  the  controller 
actively  organizes  the  constellation  of  strips.  Under 
automation  any  subjective  organization  may  be  elimi¬ 
nated  or  made  more  difficult.  Certainly  under  the 
automation-analog  we  consider  here  the  controller 
had  no  impact  on  the  organization  of  the  FPSs. 

Retrospective  Memory 

Retrospective  memory  is  memory  for  past  informa¬ 
tion  and  actions  (for  a  recent  review  see  Baddeley, 
1990).  The  importance  of  past  episodic  events  (e.g., 
clearances,  the  history  of  the  flight,  presence  ofweather) 
suggests  that  good  retrospective  memory  will  facilitate 
controller  performance  in  numerous  ways  (for  a  review 
see  Vingelis,  Schaeffer,  Stringer,  Gromelski,  &  Ahmed, 
1990). 
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Direct  involvement.  Memory  is  better  for  informa¬ 
tion  that  a  person  generates  or  action  a  person  takes,  as 
opposed  to  information  a  person  perceives  or  actions 
that  a  person  observes  being  done  (e.g.,  Slamecka  & 
Graf,  1978).  In  the  current  system,  from  the  time  that 
an  FPS  is  printed  to  the  time  that  it  is  removed  from 
the  bay,  the  controller  is  directly  interacting  with  the 
FPS.  The  nature  of  the  controller’s  direct  interaction 
with  the  strip  is  thought  to  result  in  improved  inciden¬ 
tal  memory  for  flight  information:  “The  controller 
might  offset  a  particular  flight  strip  within  the  flight 
strip  board  as  a  memory  aid.  The  fact  that  the  action 
was  under  the  controller’s  own  initiative  helped  the 
controller  recall  why  it  had  been  taken  and  what  had  to 
be  remembered.”  (Hopkin,  1989,  p.  1639).  Under 
automation,  many  of  these  operations  will  be  done  by 
computer. 

Organization.  Placing  a  strip  at  a  particular  place  in 
the  bay  when  a  flight  enters  the  controller’s  sector 
enhances  organizational  and  relational  aspects  of 
memory  for  the  flight  data  (Bower,  1 970;  Mandler, 
1961)  by  emphasizing  similarities  among  aircraft  (e.g., 
going  to  the  same  destination,  at  the  same  altitude). 
Also  important  is  the  encoding  of  distinctive  informa¬ 
tion;  for  example,  noticing  that  a  plane  is  behind 
schedule  or  at  a  lower  than  expected  altitude.  Under 
automation,  the  computer  will  automatically  place 
strips  in  the  bay,  possibly  impairing  the  encoding  of 
relational  and  distinctive  information. 

Furthermore,  the  controller  is  currently  responsible 
for  reorganizing  the  order  of  FPSs  to  reflect  the  chang¬ 
ing  conceptual  structure  of  the  traffic  situation.  Under 
automation,  much  of  this  reorganization  will  be  done 
for  the  controller.  Organization  has  perhaps  the  great¬ 
est  potential  for  improving  memory  performance,  and 
thus  any  reduced  amount  for  harming  memory  perfor¬ 
mance. 

Cuing.  Gromelski,  Davidson,  and  Stein  (1992)  col¬ 
lected  and  cataloged  controller  memory  aids.  The 
three  memory  aids  used  most  often  all  involved  strips: 
1)  arrangement  of  strips  in  a  logical  manner;  2)  offset¬ 
ting  strips;  3)  marking.  It  may  prove  difficult  to  use  the 
electronic  equivalent  of  the  strips  as  a  memory  aid  or 
cue  that  is  cognitively  equivalent  to  paper  FPSs. 

Intons-Peterson  and  Fournier  (1986)  studied  the 
use  and  effectiveness  of  external  and  internal  memory 


aids.  External  memory  aids  are  preferred  for  spatial 
tasks,  in  our  case,  presumably,  for  the  position  of 
aircraft  on  the  two  dimensional  PVD.  Although  “pa¬ 
per  strip  equivalents”  will  b*^  available  in  the  auto¬ 
mated  system,  lost  functionality  may  make  them  less 
effective  external  cues. 

In  fact,  an  experiment  by  Lansdale,  Simpson,  and 
Stroud  ( 1 990)  suppons  exactly  this  idea.  Lansdale  et 
al.  had  subjects  perform  a  filing  task  in  which  each  item 
of  information  was  to  be  annotated  for  later  retrieval. 
These  annotations  were  either  generated  by  the  com¬ 
puter  or  composed  by  the  subject.  The  latter  condition 
led  to  improved  memory.  By  analogy,  “marking”  on 
electronic  strips  will  be  much  more  restrictive  than 
marking  on  paper  strips.  Extrapolating  the  results  of 
the  Lansdale  et  al.  study  suggests  that  this  restriction 
may  hamper  memory  by  reducing  the  effectiveness  of 
the  external  cues. 

Prospective  Memory  and  Planning 

In  addition  to  ^cilitating  the  recollection  of  past 
events,  it  has  been  argued  that  flight  strips  support 
prospective  memory  through  their  use  as  external 
memory  aids  (Vortac  &  Gettys,  1991).  Prospective 
memory  is  the  use  of  memory  to  remember  to  take  a 
planned  action  at  a  future  time.  This  is  essential  in  air 
traffic  control,  with  controllers  having  to  remember  to 
return  to  earlier  clearance  requests,  or  having  to  con¬ 
form  to  plans  made  earlier. 

In  comparison  to  retrospective  memory,  prospec¬ 
tive  memory  has  been  little  researched  (Harris,  1984), 
although  that  is  beginning  to  change,  especially  in 
light  of  the  results  that  show  that  the  two  types  of 
memory  are  uncorrelated  (e.g.,  Einstein  &  McDaniel, 
1990;Kvavilashvili,  1987;  Meacham  &  Leiman,  1975; 
Wilkins  &  Baddeley,  1978;  but  see  Hitch  &  Ferguson, 
1991).  In  fact,  some  of  the  work  is  quite  provocative 
from  the  perspective  of  the  current  paper:  For  ex¬ 
ample,  the  Intons-Peterson  and  Fournier  (1986)  study 
cited  earlier  showed  that  external  memory  cues  are 
preferred  for  future  remembering. 

Distinctiveness.  Distinctiveness  or  unfamiliarity  of 
an  external  cue  has  been  found  to  result  in  superior 
prospective  memory  performance  (Einstein  & 
McDaniel,  1990).  A  paper  strip  is  distinctive  either  by 
the  amount  and  kind  of  idiosyncratic  marking  done  by 
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the  controller  or  by  being  physically  offset  from  the 
others  in  the  bay.  Because  automation  will  force  stan- 
dardization  and  thereby  lessen  distinctiveness  (as  dis¬ 
cussed  earlier),  the  electronic  equivalents  of  paper 
FPSs  may  not  be  as  distinctive  and  may  therefore  be  a 
relatively  poor  prospective  memory  cue. 

Motoric  enactment.  Koriat,  Ben-Zur,  and  Nussbaum 
(1990)  found  that  motoric  enactment  of  future  actions 
(e.g.,  lift  one  object,  replace  another)  improved  pro¬ 
spective  memory,  compared  to  verbal  rehearsal.  A 
conceivable  implication  is  that  the  automated  system, 
which  cannot  support  any  distinct  motoric  compo¬ 
nent  because  input  is  done  through  the  keyboard  or 
trackball,  may  support  prospective  memory  less  effi¬ 
ciently  than  the  current  paper  strip  system. 

Method 

Experimental  Automation  Analog 

The  FPSs  used  in  the  experimental  (Restricted) 
condition  showed  only  4  of  the  usual  3 1  pieces  of  flight 
data:  call  sign,  aircraft  type,  assigned  altitude,  and 
route.  In  addition,  the  strip  holders  were  glued  to¬ 
gether  and  subjects  were  not  permitted  to  touch, 
move,  write  on,  or  manipulate  them  in  any  way.  The 
Restricted  condition  was  compared  to  the  Normal 
condition,  in  which  subjects  had  full  access — includ¬ 
ing  writing  and  manipulating — to  the  complete  cur- 
rently-used  strips. 

The  Restricted  condition  provided  an  experimental 
analog  to  ISSS  in  several  ways:  First,  the  amount  of 
information  remaining  on  the  skeleton  strips  approxi¬ 
mated  a  minimal  version  of  the  electronic  displays 
currently  under  consideration.  Second,  the  less  direct 
interactive  nature  of  the  electronic  display  was  mim¬ 
icked  by  restricting  the  controller’s  ability  to  move 
strips.  Other  methods  for  retrieving  flight  data  avail¬ 
able  on  the  system  were  left  intact.  For  example, 
controllers  had  the  option  of  retrieving  further  flight 
information  from  the  computer  via  additional  key¬ 
board  interaction. 


Subjects 

A  total  of  20  subjects  participated,  with  1 0  subjects 
randomly  assigned  to  each  group.  All  subjects  were 
instructors  at  the  FAA  Academy,  and  comprised  a 
nearly  exhaustive  sample  of  that  population.  All  were 
full-performance  level  controllers  who  had  been  en 
route  controllers  for  an  average  of  6.7  years,  and  last 
served  in  the  field  17  months  prior  to  the  study.  Eight 
of  the  subjects  had  participated  in  an  earlier  observa¬ 
tional  study  conducted  by  the  same  research  team 
(Vortac,  Edwards,  Jones,  Manning,  &  Rotter,  in  press). 

Each  subject  participated  individually  in  two  sce¬ 
narios  of  medium  complexity.  The  first  scenario  was 
used  to  obtain  performance  measures.  The  second 
scenario  was  run  in  two  segments,  separated  by  a 
battery  of  cognitive  tasks. 

Materials 

The  experiment  was  conducted  at  the  en  route 
Radar  Training  Facility  (RTF)  at  the  FAA’s  Mike 
Monroncy  Aeronautical  Center  in  Oklahoma  City, 
which  provides  high-fidelity  air  traffic  simulations 
using  the  fictitious  AeroCenter  airspace  used  in  Acad¬ 
emy  training.  All  subjects  were  instructors  familiar 
with  AeroCenter. 

Two  experimenters  were  present  throughout  each 
experimental  session.  In  addition,  a  subject  matter 
expert  (acting  manager  of  the  RTF  or  a  designated 
substitute)  was  present  during  the  first  scenario  to 
assess  controller  performance  and  to  assume  the  role  of 
supervisor.  Two  “ghost  pilots”  controlled  the  planes, 
and  another  ghost  assumed  the  communication  func¬ 
tions  of  adjacent  Centers  and  other  ATC  facilities. 

Two  scenarios  of  medium  complexity  (6.5  depar¬ 
tures,  10.5  arrivals,  and  11.5  overflights  on  average) 
unfamiliar  to  the  Academy  instructors  were  selected 
with  the  help  of  our  subject  matter  expert.  Each 
scenario  lasted  approximately  30  minutes  and  involved 
levels  of  traffic  density  that  in  the  field  could  be 
handled  by  an  individual. 


5 


Auumation  and  Captitian 


Procedure 

Prior  to  a  subject’s  arrival  at  the  RTF,  all  FPSs  were 
placed  in  the  strip  bay  (located  to  the  right  of  the  radar) 
sorted  in  ascending  order  by  time  of  arrival  or  depar¬ 
ture.  The  strip  bay  was  then  covered. 

Subjects  first  completed  a  background  question¬ 
naire.  The  subject  matter  expert  then  familiarized 
participants  with  the  letters  of  agreement,  handoff 
procedures,  and  radio  frequencies  relevant  to  the  sce¬ 
nario.  All  subjects  were  familiar  with  AeroCenter,  but 
some  had  been  accustomed  to  controlling  AeroCenter 
traffic  using  non-radar  procedures  and  needed  famil¬ 
iarization  with  specific  procedures  for  radar  ATC. 

Instructions  were  then  given  regarding  the  Normal 
or  Restricted  condition.  The  normal  group  was  given 
5  minutes  to  set  up  and  inspect  the  FPSs,  and  the 
restricted  group  received  the  same  amount  of  time  to 
inspect  the  FPSs.  The  Normal  group  was  told  they 
could  arrange  and  write  on  the  FPSs  as  they  normally 
would  in  the  field;  the  Restricted  group  was  told  they 
could  only  look  at  the  (limited)  FPSs.  The  scenario 
started  when  subjects  indicated  that  they  had  in¬ 
spected  the  strip  bay  to  their  satisfaction,  or  when  the 
maximum  time  of  5  minutes  had  elapsed. 

During  the  scenario,  an  audio  record  was  kept  of  the 
controller  and  the  ghost  pilots.  The  subject’s  interac¬ 
tions  with  the  Quick  Action  Keyboard  (QAK)  on  the 
radar  console  were  recorded.  Some  of  the  keys  on  the 
QAK  can  be  used  to  replace  FPS  functionality:  For 
example,  one  key  provides  a  display  of  an  aircraft’s 
route  on  the  radar,  and  another  one  prints  out  the 
complete  flight  plan  on  the  Computer  Readout  Dis¬ 
play  (CRD;  a  small  screen  next  to  the  radar).  The 
frequency  with  which  these  keys  arc  pressed  thus 
provides  an  index  of  the  extent  to  which  a  subject 
replaced  an  interaction  with  a  flight  strip  with  a  com¬ 
puter  interaction.  In  addition,  during  the  scenario,  the 
experimenter  recorded  the  number  of  commands  con¬ 
trollers  issued  and  the  number  of  requests  made  by  the 
controllers.  These  two  controller  activities  were  in¬ 
tended  to  provide  a  measure  of  any  gross  differences  in 
how  the  groups  differed  in  their  interaction  with  the 
air  traffic. 


Scenario  I  (Performance  Measures) 

Assessment  of  ATC  performance  is  a  non-trivial 
task.  Buckley,  DcBaryshe,  Hitchner,  and  Kohn  (1983) 
examined  some  28  possible  indices,  among  them  fuel 
consumption,  delays,  and  subjective  variables  (e.g., 
performance  rating  by  subject  matter  experts).  Buckley 
et  al.  suggested  that  a  combination  of  both  subjective 
and  objective  measures  is  required  for  a  full  perfor¬ 
mance  assessment.  Hence,  the  present  experiment  in¬ 
cluded  (i)  standard  “over-the-shoulder”  evaluation  by 
a  subject  matter  expert,  (ii)  on-line  evaluation  of  a 
position  relief  briefing  by  the  same  expert,  and  (iii)  a 
quasi-objective  post-scenario  analysis  of  the  traffic 
pattern. 

“Over-the-shoulder" evaluation.  “Over-the-shoulder” 
evaluation  was  conducted  by  the  subject  matter  expert, 
using  the  FAA’s  On  the  Job  Training  evaluation  form 
(OJT;  FAA  Form  3120-25).  The  OJT  form  has  several 
items  directed  at  five  aspects  of  ATC  performance: 
Separation,  control  judgment,  methods/procedures, 
equipment,  and  communication/coordination. 

Position  relief  briefing.  Twenty-one  minutes  into 
the  first  scenario,  the  subject  matter  expert  requested 
an  unanticipated  position-relief  briefing.  The  posi¬ 
tion-reliefbriefing  was  conducted  as  in  the  field  when 
one  controller  provides  information  relevant  for  the 
relieving  controller  to  take  over  the  control  of  the 
sector.  The  quality  of  the  briefing  was  assessed  using  a 
position-relief  briefing  checklist  which  contained  sev¬ 
eral  items  that  queried  the  completeness  of  actions 
dealing  with  traffic  and  non-traffic  aspects  of  the 
current  situation. 

Post-scenario  analysis.  The  scenario  was  terminated 
at  27  minutes,  and  after  the  subject  had  left  the 
laboratory,  the  subject  matter  expert  analyzed  the 
traffic  pattern  and  decided,  for  each  aircraft  on  the 
PVD,  how  many  route,  speed,  or  altitude  changes  (if 
any)  remained  to  be  taken  to  get  the  plane  out  of  the 
controller’s  sector.  Given  the  same  set  of  initial  condi¬ 
tions  and  a  constant  amount  of  elapsed  time,  the  more 
efficient  controller  should  have  fewer  control  actions 
remaining  for  a  given  aircraft  than  the  less  efficient 
controller.  We  consider  this  a  quasi-objective  measure 
because,  for  a  subject  matter  expert,  there  is  little 
uncertainty  about  what  actions  are  required  for  a  given 
aircraft  before  it  is  handed  over  to  an  adjacent  facility. 
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Scenario  II  (Cognitive  Measures) 

The  two  scenarios  were  separated  by  a  1 5-minute 
break  period.  Subjects  followed  the  same  bay  inspec¬ 
tion  procedure  for  the  second  scenario  and  no  addi¬ 
tional  instructions  were  given.  The  prospective  memory 
and  attention  measures  were  taken  unobtrusively  dur¬ 
ing  the  scenario. 

Prospective  Memory.  Prospective  memory  was  tested 
by  incorporating  pilot  requests  to  which  the  controller 
could  not  immediately  respond.  On  three  separate 
occasions,  pilots  made  a  request  before  their  aircraft 
had  entered  the  controller’s  airspace.  Because  the  air¬ 
craft  were  then  still  under  the  adjacent  sector’s  control, 
the  controller  had  to  tell  the  pilot  to  standby  and  had 
to  remember  to  reply  to  the  initial  request  after  the 
plane  had  entered  the  sector.  (Another  legal  option  for 
the  controller  would  have  been  to  call  the  adjacent 
sector  for  permission  to  get  control  immediately,  in 
order  to  deal  with  the  situation  on  the  spot.  These 
requests  were  always  denied.)  Specifically,  at  3:02 
minutes  into  the  scenario,  a  general  aviation  flight 
requested  to  be  vectored  around  weather;  at  9:30  a 
commercial  flight  requested  a  lower  altitude;  and  at 
18:15  a  military  flight  requested  direct  routing  to 
Kansas  City. 

Attentional  Engagement  and  Visual  Search.  Seven 
measurements  of  attentional  load  were  obtained  dur¬ 
ing  the  scenario  by  ringing  the  land  line  (telephone) 
from  an  adjacent  center.  Time  was  measured,  using  the 
audio  record,  from  the  first  ring  until  the  subject 
answered.  On  two  occasions,  when  the  controller 
answered  the  land  line  he  or  she  was  asked  whether  an 
FPS  for  a  particular  aircraft  was  present.  In  both  cases, 
the  strip  was  absent,  necessitating  an  exhaustive  search 
of  the  strip  bays.  The  subsequent  latency  to  scan  the 
strip  bay — defined  as  the  delay  to  answer  the  request — 
provided  an  index  of  visual  search  times. 

At  the  16-minute  mark,  the  scenario  was  tempo¬ 
rarily  suspended.  At  this  point,  13  aircraft  were  visible 
on  the  PVD:  five  commercial,  five  general  aviation, 
and  three  military.  Three  of  these  had  very  recently 
appeared  on  the  screen  and  were  not  yet  in  the 
controller’s  sector.  The  controller  was  instructed  to 
turn  away  from  the  radar  and  the  remaining  battery  of 
cognitive  tests  was  administered. 


Map  Recall.  The  controller  was  instructed  to  recall 
the  positions  (with  call  signs  if  possible),  of  all  con¬ 
trolled  aircraft  by  marking  the  appropriate  locations 
on  a  clear  sheet  of  plastic.  The  sheet  contained  sector 
boundaries,  jetways,  and  other  landmarks  used  on  the 
radar,  as  shown  in  Figure  2.  Six  minutes  were  allowed 
for  Map  Recall. 

Cued  FPS  Recall.  Subjects  were  then  given  a  test 
booklet  containing  one  FPS  to  a  page,  with  only  the 
call  sign  provided.  FPSs  were  arranged  in  a  different 
random  order  for  each  subject,  and  recall  of  three  fields 
on  each  strip  (aircraft  type,  altitude,  and  route)  was 
cued  by  a  ’?’  symbol.  Subjects  were  instructed  to 
proceed  through  the  booklet  in  order,  without  skip¬ 
ping  pages  or  returning  to  previous  strips  and  to 
complete  as  many  of  the  cued  fields  as  possible.  They 
were  given  10  minutes  to  complete  this  task.  Upon 
completion  of  the  first  pass  through  the  test  booklet, 
subjects  were  given  a  different  color  pen  and  asked  to 
recall  any  additional  information  they  might  remem¬ 
ber  about  the  flights.  Subjects  co  li  '  work  through  the 
booklet  in  any  order  on  the  second  pass.  They  were 
given  six  additional  minutes  to  complete  this  task. 

Traffic  Planning.  Finally,  subjects  were  turned  back 
toward  the  PVD  and  allowed  to  look  at  the  radar  and 
the  strips.  The  PVD  was  covered  with  a  clear  sheet  of 
plastic  and  subjects  were  asked  to  draw  the  anticipated 
exact  flight  path  for  all  flights  for  the  next  1 0  minutes. 
They  were  told  that  the  scenario  would  be  re-run  after 
the  experiment  to  measure  conformance  between  ac¬ 
tual  and  planned  flight  paths.  Upon  completion  of  the 
traffic  planning  task,  the  scenario  resumed  for  the 
remaining  14  minutes  without  any  further  interrup¬ 
tions. 

Results 

Analysis  of  each  set  of  measures  began  with  an 
omnibus  multivariate,  Pillai’s  trace  (MANOVA)  or 
univariate  analysis  of  variance  (ANOVA);  significant 
omnibus  tests  justified  subsequent  univariate  tests  and 
contrasts.  All  tests  used  a  =  .05  unless  otherwise 
indicated.  Measures  of  performance  and  control  ac¬ 
tions  were  from  the  uninterrupted  first  scenario;  mea¬ 
sures  of  cognition  and  flight  strip  use  were  from  the 
interrupted  second  scenario. 
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Control  actions 

A  MANOVA  indicated  that  controllers  with  re¬ 
stricted  access  to  limited  strips  were  similar  to  control¬ 
lers  with  normal  access  to  the  strips  in  the  number  of 
verbal  commands  issued  (M  =  57.8  and  52.6,  for 
Normal  and  Restricted,  respectively)  and  in  the  num¬ 
ber  of  requests  uttered  (M  =  11.5  and  8.0,  for  Normal 
and  Restricted,  respectively). 

The  frequency  of  QAK  keys  pressed  (i.e.,  flight  plan 
readout,  assigned  altitude,  interim  altitude,  and  route 
projection)  were  analyzed  for  nine  subjects  in  each 
condition.  (Scenario  summaries  for  one  subject  in  each 


condition  were  1  jst  due  to  equipment  malfunction.)  A 
MANOVA  [f  (4, 13)= 1 .55]  showed  that  the  two  groups 
of  controllers  made  similar  use  of  the  QAK  (see  Table 
1 ).  The  apparently  large  difference  between  the  groups 
on  the  use  of  the  flight  plan  readout  key  was  caused  by 
one  subject  in  the  Restricted  condition  using  the  key 
over  25  times,  inflating  the  mean  and  variability  for 
that  group.  Thus,  both  in  terms  of  t*'e  frequency  of 
commands  and  in  the  controller’s  interactions  with  the 
QAK,  the  restriction  placed  on  the  strips  did  not 
reliably  alter  interaction  with  the  air  traffic  or  the 
computer.^ 


Figure  2.  Sector  map  used  during  Map  Recall.  Subjects  marked  the 
recalled  locations  of  all  aircraft  together  svith  their  call  sign. 


^  All  analyses  reported  in  this  article  were  conducted  without  the  subject  in  the  restricted  condition  who  appeared  to  compensate  for 
the  limited  strips  by  using  the  QAK  and  without  the  subject  in  the  normal  condition  who  chose  not  to  mark  the  FPS's.  We  thought  that 
including  these  subjects  might  reduce  the  difference  b^een  the  experimental  condition  and  the  control  condition,  thus  masking  a 
possible  effect  of  restricting  access.  The  additional  analyses  supplied  no  support  for  this  concern,  and  thus  we  report  only  those  analyses 
that  included  all  of  the  subjects. 
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Table  1 

Frequency  of  quidc  action  key  presses 


Condition _ Key _ Mean _ Std.  Deviation 


Normal 

Flight  Plan 

1.78 

2.86 

Assigned  Altitude 

5.67 

3.04 

Interim  Altitude 

10.67 

8.79 

Route 

13.56 

9.19 

Restricted 

Flight  Plan 

6.33 

8.93 

Assigned  Altitude 

4.78 

3.23 

Interim  Altitude 

7.89 

4.91 

Route 

8.44 

4.90 

Table  2 


Frequency  of  n^ative  performance  comments,  OJT  Form 


Condition 

Performance  Class 

Mean 

Std.  Deviation 

Normal 

Separation 

0.50 

0.53 

Control  judgment 

2.10 

1.60 

Methods/Procedures 

2.50 

1.43 

Equipment 

0.00 

0.00 

Communication/Coordination 

0.40 

0.52 

Restricted 

Separation 

0.40 

0.70 

Control  Judgment 

2.00 

1.89 

Methods/Proced  u  res 

1.50 

1.90 

Equipment 

0.10 

0.32 

Communication/Coordination 

0.80 

1.13 

Flight  Strip  Use 

To  verify  that  subjects  in  the  Normal  condition 
interacted  with  flight  strips,  the  number  of  markings 
and  changes  made  on  the  FPSs  during  the  second 
scenario  was  analyzed.  The  average  number  of  mark¬ 
ings  on  all  strips  was  126.6,  with  a  maximum  of  216 
and  a  minimum  of  1.  Hence,  only  one  subject  in  the 
Normal  group  chose  not  to  mark  on  the  FPSs  (the 
second-lowest  number  was  84),  suggesting  that  most 
controllers  in  the  Normal  condition  interacted  with 
strips  in  the  manner  expected  in  the  fleld.^ 


Performance 

OJT  Form 

The  number  of  negative  comments  made  by  the 
subject  matter  expert  was  tabulated  for  each  major 
section  of  the  OJT  form  (see  Table  2).  A  MANOVA 
indicated  that  judged  performance  under  the  Restricted 
condition  was  comparable  to  performance  under  the 
Normal  condition  [F(5,14)<1]. 
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Position-Relief Briefing 

The  proportions  of  both  traffic  and  nontraffic  items 
covered  satisfactorily  in  the  position-relief  briefing 
were  computed.  A  MANOVA  on  these  two  variables 
again  indicated  no  difference  in  performance  between 
the  Restricted  and  Normal  conditions  [f  (2,16)<l]. 
Normal  controllers  covered  74%  of  the  traffic  items 
and  44%  of  the  nontraffic  items  satisfactorily;  Re¬ 
stricted  controllers  covered  80%  of  the  traffic  items 
and  33%  of  the  nontraffic  items  satisfactorily. 

Post-Scer-  - '  j  analysis 

We  tabulated  separately  the  number  of  route,  alti¬ 
tude,  and  speed  changes  the  subject  matter  experts 
indicated  were  remaining  to  be  performed  at  the  end  of 
the  scenario.  The  MANOVA  revealed  that  controllers 
in  the  Restricted  condition  tended  to  have  more  ac¬ 
tions  yet  to  be  performed  than  did  those  in  the  Normal 
condition,  F  (3,  16)  =  2.45,  p  =  .10.  Although  the 
effect  was  not  significant  by  the  .05  criterion,  the 
importance  of  a  possible  performance  decrement  dic¬ 
tated  that  any  potential  effect,  however  slight,  was 
worthy  of  further  consideration.  Thus,  subsequent 
univariate  ANOVAs  were  conducted.  These  tests  indi¬ 
cated  that  the  difference  between  conditions  was  due 
entirely  to  the  controllers  in  the  Normal  condition 
having  initiated  more  required  route  changes  than  did 
those  in  the  Restricted  condition  [F(l,  18)  =  5.31], 
whereas  no  differences  were  found  for  required  speed 
[F(l,18)  <  1]  and  altitude  [F(l,18)  =  1.29]  changes. 
On  average,  controllers  in  the  Restricted  condition 
had  1 . 1  more  route  changes  remaining  than  controllers 
in  the  Normal  condition. 

Summary  of  Performance  Measures 

Overall,  the  performance  measures  suggest  that 

controllers  in  the  Restricted  condition  controlled  traffic 

as  well  as  those  under  Normal  conditions.  The  only 

suggestion  of  a  performance  deficit  was  a  small 

univariate  effect  in  the  number  of  route  changes 

remaining. 

Cognitive  measures 
Attentional  measures 

Data  were  analyzed  separately  for  the  time  to  answer 
the  land  line  and  the  time  to  search  the  strip  bay.  Mean 


latencies  were  computed  for  each  subject  across  the 
seven  land  line  calls.  An  AWS  r-test  (e.g.,  Winer, 
Brown,  &  Michels,  1991)  revealed  no  difference  be¬ 
tween  Normal  (M  =  23. 1  s)  and  Restricted  (M  =  27. 7  s), 
suggesting  that  attentional  demands  of  controlling 
traffic  were  comparable  under  the  two  conditions. 

Mean  latencies  to  deny  the  existence  of  the  re¬ 
quested  strip  were  computed  for  each  subject  across 
the  two  opportunities.  These  latencies  were  corrected 
for  the  number  of  strips  currently  in  the  bay  when  the 
request  was  made,  yielding  a  time  per  strip.  (This  was 
necessary  because  the  Normal  group  could  remove 
unnecessary  strips  and  so  had  fewer  through  which  to 
search).  Again,  the  AWS  t  revealed  no  difference  be¬ 
tween  the  two  conditions,  indicating  that  control¬ 
lers  could  search  a  strip  bay  they  could  not  manipulate 
(M  =  .624  s/strip)  as  quickly  as  they  could  search  a  strip 
bay  they  had  organized  (M  =  .603  s/strip). 

Retrospective  memory 

Two  datasets  were  created  to  investigate  retrospec¬ 
tive  memory.  One  dataset  drawn  from  Map  Recall 
comprised  the  number  of  aircraft  correctly  placed  on 
the  sector  map  and  the  missed-distance  (in  cm)  of 
those  placements  (see  Table  3).  The  other  dataset  from 
Cued  FPS  Recall  comprised  the  proportion  of  correct 
recall  averaged  across  all  relevant  strips  for  each  sub¬ 
ject,  of  the  aircraft  type,  altitude  upon  sector  entry, 
route,  and  destination  (see  Table  4).  Only  one  subject 
added  additional  flight  data  on  the  second  attempt, 
and  thus  only  the  first  attempt  at  recall  was  analyzed. 
In  addition  to  comparing  Normal  and  Restricted  ac¬ 
cess,  these  MANOVAs  divided  aircraft  into  commer¬ 
cial  (there  were  five  on  the  screen),  military  (three), 
and  general  aviation  (five). 

The  analysis  of  the  Map  Recall  dataset  failed  to 
reveal  a  difference  between  conditions,  with  those  in 
the  Normal  condition  recalling  30%  of  the  aircraft 
with  their  proper  call  sign  and  those  in  the  Restricted 
condition  recalling  44%  of  the  aircraft  with  their  call 
signs  [F  (2,17)=1.57].  The  low  absolute  level  of 
performance  was  due  to  the  requirement  that  the 
correct  call  sign  be  affixed  to  a  target  placed  on  the 
sector  map.  When  targets  are  counted  without  regard  to 
accuracy  of  call  sign,  the  recall  levels  were  63%  and  75% 
for  Normal  and  Restricted,  respectively.  Although 
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Table  3 

Map  Recall  Performance 


Condition  Type  of  Aircraft  Proportion  Recalled  Missed  Distance  (cm) 


Mean 

Std.  Dev. 

Mean 

Std.  Dev. 

Normal 

Commercial 

.50 

.30 

2.17 

1.25 

General  Aviation 

.08 

.17 

3.83 

2.23 

Military 

.33 

.00 

2.02 

1.24 

Restricted 

Commercial 

.64 

.35 

2.25 

2.04 

General  Aviation 

.28 

.23 

2.17 

1.65 

Military 

.40 

.21 

3.16 

3.38 

Table  4 

FPS  Recall  Performance 


History  of  Aircraft 

Condition 

Type  of  Aircraft 

Mean  Proportion  Recalled 

A/C  Type  Altitude  Route  Destination 

Currently  on  PVD 

Normal 

Commercial 

0.12 

0.38 

0.12 

0.42 

General  Aviation 

0.05 

0.04 

0.12 

0.14 

Military 

0.11 

0.28 

0.10 

0.03 

M 

0.09 

0.23 

0.11 

0.20 

Restricted 

Commercial 

0.12 

0.42 

0.18 

0.48 

General  Aviation 

0.07 

0.06 

0.12 

0.14 

Military 

0.11 

0.26 

0.03 

0.10 

M 

0.10 

0.25 

0.11 

0.24 

No  longer  on  PVD 

Normal 

Commercial 

0.00 

0.13 

0.23 

0.23 

General  Aviation 

0.03 

0.00 

0.13 

0.07 

Military 

0.06 

0.36 

0.23 

0.00 

M 

0.03 

0.17 

0.20 

0.10 

Restricted 

Commercial 

0.07 

.13 

0.23 

0.30 

General  Aviation 

0.00 

0.00 

0.13 

0.07 

Military 

0.06 

0.36 

0.17 

0.00 

M 

0.04 

0.17 

0.18 

0.12 
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insignificant,  it  is  worth  noting  that  recall  is  in  a 
direction  opposite  that  expected  by  those  concerned 
that  less  interaction  with  the  strips  would  reduce 
memory. 

The  type  of  aircraft  had  a  large  overall  effect;  F(4, 72)  = 
6.45.  Exploration  of  the  effect  by  multiple  correlated- 
groups  r-tests  revealed  that  commercial  planes  were 
recalled  better  than  military,  r(18)  =  2.90,  or  general 
aviation  aircraft,  /(1 8)  =  6.09,  and  military  were  recalled 
better  than  general  aviation,  t  (18)  =  4.03.  This  effect 
appeared  only  in  the  number  of  call  signs  placed  on  the 
map,  not  in  the  missed-distance  of  the  recalled  flights 
[^■(2,38)  <  1].  The  effect  of  flight  class  on  memory  is 
consistent  with  the  preponderance  of  commercial 
flights,  but  the  absence  of  an  interaction  of  condition 
with  flight-class  also  indicates  that  limited  access  to  the 
strips  did  not  affect  this  relative  ordering. 

Unlike  the  Map  Recall  data,  the  Cued  EPS  Recall 
included  planes  that  had  not  yet  appeared  on  the  PVO 
and  planes  that  had  left  the  PVD,  as  well  as  those 
planes  curren  tly  visible.  We  performed  a  separate  analy¬ 
sis  on  each  set  of  strips.  Recall  of  flight  data  for  future 
flights  was  essentially  zero  for  all  controllers.  For  past 
and  current  flights  there  were  no  differences  between 
Restricted  and  Normal  access  to  strips.  However,  as 
with  Map  Recall,  the  class  of  flight  did  yield  significant 
multivariate  Fs  (current  flights:  F(8,68)  =  9.32,  and 
past:  F(8, 68)  =  9.67].  Recall  of  the  flight’s  destination 
was  better  for  commercial  than  military  flights  (f(18) 
=  6.84  for  past  and  t  (18)  =  6.03  for  current)  and 
general  aviation  flights  (f(18)  =  3.94  for  past  and  f(18) 
=  4.41  for  current).  Destinations  of  general  aviation 
flights,  in  turn,  were  recalled  better  than  those  of 
military  flights  (r(18)  =  2.18for  past  and  r(18)s2.10 
for  current) .  Again,  limited  access  to  flight  data  did  not 
affect  the  ordering  between  classes  of  flights  suggesting 
that  retrospective  memory  may  be  relatively  immune 
to  the  types  of  automation  changes  captured  here. 

Prospective  Memory 

The  prospective  memory  data  (see  Figure  3)  com¬ 
prised  the  number  of  pilot  requests  that  were  ulti¬ 
mately  granted,  as  well  as  the  delay  before  those  requests 
were  granted.  The  MANOVA  showed  reliable  differ¬ 
ences  between  groups,  F(2,  17)  =  4.34,  and  the  subse¬ 
quent  ANOVAs  showed  that  the  effect  was  due  to 


controllers  in  the  Restricted  condition  remembering 
to  grant  more  requests,  t  (18)  -  2.35,  and  tending  to 
grant  them  sooner  than  controllers  in  the  Normal 
condition,  f  (18)  =  1.93,^=  .07. 

Planning 

The  planning  data  (see  Table  5)  comprised  the 
number  of  turns  executed  (or  not  executed)  during  the 
10  minutes  following  resumption  of  the  second  sce¬ 
nario,  compared  to  the  number  of  turns  previously 
planned.  The  planning  data  were  based  on  five  aircraft 
for  each  subject.  Three  of  these  aircraft  had  just  ap  ¬ 
peared  on  the  PVO  and  were  about  to  enter  the 
controller’s  sector.  Of  all  the  other  flights  on  the  PVD, 
only  two  clearly  required  route  changes  or  turns  within 
the  next  10  minutes.  For  these  five  planes,  we  first 
determined  if  the  subject  had  correctly  indicated  the 
true  heading,  plus  or  minus  30  degrees.  This  restric¬ 
tion  ensured  that  the  starting  point  for  the  controllers’ 
plans  and  the  actual  state  of  the  airspace  were  similar. 
This,  in  turn,  allowed  us  to  match  planned  turns  with 
turns  actually  ordered  when  the  scenario  resumed. 
This  criterion  yielded  a  total  of  35  and  36  usable 
observations  for  the  Normal  and  Restricted  condition, 
respectively,  out  of  a  total  possible  of  50  for  each 
condition  (5  flights  x  10  subjects).  The  classifications 
into  possible  outcomes  is  shown  in  Table  5. 

As  with  prospective  memory,  the  planning  data  can 
also  be  viewed  to  be  suggestive  of  a  Restricted-condi¬ 
tion  superiority.  An  independent-groups  ^test  indi¬ 
cated  that  subjects  in  the  Restricted  condition 
anticipated  significantly  more  (80%:  16/20)  of  subse¬ 
quently  made  turns  than  in  the  Normal  condition 
(52%:  14/27;  f(18)  =  3.12). 

Alternatively,  focus  may  rest  on  the  fact  that  the 
Normal  controllers  were  more  likely  to  change  the 
route  of  aircraft  for  which  their  initial  plan  failed  to 
foresee  any  such  changes  (65%:  13/20;  see  Tabic  5). 
This  compares  to  the  Restricted  condition,  where 
controllers  performed  considerably  fewer  such  turns 
(24%:  4/17).  Assuming  that  both  conditions  ulti¬ 
mately  required  the  same  number  of  turns,  the  differ¬ 
ence  may  suggest  that  the  Normal  controllers  detected 
necessary  route  corrections  more  quickly  than  the 
Restricted  subjects.  Note  that  this  analysis  is  consis¬ 
tent  with  the  effect  observed  in  the  post-scenario 
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Figure  3.  Prospective  memory  performance  across  the  two  conditions. 
Latency  to  comply  with  request  is  shown  on  the  left-hand  axis  and 
accuracy  (proportion  compliance)  on  the  right-hand  axis. 


Tables 

Frequency  of  Conformance  and  Non-conformance  to  Planned  Actions 


Condition 

Action  Planned 

Turned 

Action  Taken 

Not  Turned 

Normal 

Turned 

14 

1 

15 

Not  Turned 

13 

7 

20 

Total 

27 

8 

Restricted 

Turned 

16 

3 

19 

Not  Turned 

4 

13 

17 

Total 

20 

16 

13 
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performance  measure  (Scenario  I),  where  controllers 
in  the  Restricted  condition  had  approximately  one 
more  route  change  remaining  at  the  end  of  the  sce¬ 
nario.  Thus,  although  the  greater  proportion  of  antici¬ 
pated  turns  speaks  to  better  planning  by  controllers  in 
the  Restricted  condition,  this  apparent  superiority  is 
accompanied  by  a  tendency  for  Restricted-condition 
controllers  to  delay  rerouting  of  some  aircraft. 

Summary  of  cognitive  measures 

Only  prospective  measures  of  cognitive  functioning 
showed  a  difference  between  normal  and  limited  ac¬ 
cess  to  FPSs,  and  those  differences  arguably  favored 
the  Restricted  condition.  The  other  measures  confirm 
that  this  advantage  of  the  Restricted  condition  was  not 
due  to  a  deficit  elsewhere:  The  Restricted  condition 
allowed  comparably  fast  access  to  the  land  lines,  equally 
speedy  search  of  the  strip  bay,  equivalent  retrospective 
memory  of  aircraft  positions,  and  comparable  retro¬ 
spective  memory  of  the  flight  data. 

Discussion 

Overall,  contrary  to  the  reasoning  and  literature 
cited  at  the  outset,  and  contrary  to  speculations  by 
aviation  experts,  reducing  access  to  flight  progress 
strips — in  a  manner  more  extreme  than  anything  fore¬ 
seen  for  ISSS — had  little  adverse  effect  on  perfor¬ 
mance,  and  no  measurable  detrimental  effect  on  the 
cognitive  measures.  In  fact,  the  only  effects  reliable  at 
conventional  levels  of  significance  involved  forward- 
looking  processes,  prospective  memory  and  planning, 
and  favored  the  restricted  condition. 

It  is  true,  however,  that  given  the  potentially  cata¬ 
strophic  impact  of  even  slight  performance  decre¬ 
ments  in  air  traffic  control,  failure  to  reach  conventional 
levels  of  significance  should  not  deter  other  researchers 
from  investigating  further  the  possible  negative  impact 
of  automation.  Specifically,  the  suggestive  result  that 
controllers  in  the  Restricted  condition  delayed  neces¬ 
sary  route  changes  (Scenario  I)  must  be  taken  seri¬ 
ously.  The  effect  was  also  arguably  reflected  in  the 
planning  data  during  the  second  scenario.  In  further 
support,  consider  the  fact  that  pragmatic  constraints 
limited  the  number  of  subjects  to  1 0  in  each  condition, 
which  may  have  prevented  any  subtle  performance 


differences  from  reaching  greater  statistical  reliability. 
On  the  other  hand,  the  restricted  condition  was  quite 
novel  to  our  subjects  and  thus  this  deficit  may  be  due 
to  the  novelty  of  the  condition  and  not  to  the  strip 
configuration.  Also  in  line  with  this  position  is  that 
any  arguments  based  on  lack  of  power  must  recognize 
that  the  nonsignificant  effects  do  not  tend  to  lean  in 
favor  of  the  normal  condition  as  a  power  argument 
would  imply.  Overall,  the  performance  data  may  be 
best  interpreted  as  sounding  a  note  of  caution  concern¬ 
ing  the  impending  removal  of  flight  strips  that  is 
worthy  of  further  exploration. 

As  for  the  cognitive  data,  lack  of  statistical  power 
cannot  be  invoked  to  explain  the  observed  beneficial 
effects  of  restricted  strip  access  on  prospective  memory. 
Restricted  subjects  were  more  likely  to  remember  to 
grant  a  previously-delayed  request,  and  complied  with 
these  requests  sooner,  than  did  controllers  with  nor¬ 
mal  access  to  the  strips. 

What  might  explain  the  superiority  of  the  Re¬ 
stricted  group?  We  propose  two  related  reasons,  in¬ 
volving  (i)  a  reduction  in  workload,  and  (ii)  a  change 
in  the  structure  of  the  task  that  induced  a  more 
strategic  behavior. 

Given  the  prospective  memory  literature  and  the 
data  and  thinking  suggesting  that  FPSs  were  useful 
external  memory  aids  (e.g.,  Gromelski  et  al.,  1992; 
Vingelis  et  al.,  1990;  Vortac  &  Gettys,  1991)  we 
expected  the  Normal  group  to  have  shown  better 
prospective  memory  than  the  Restricted  group.  How¬ 
ever,  normal  use  of  the  strips  carries  with  it  the  respon¬ 
sibility  of  performing  the  required  strip  markings  and 
physical  manipulations.  Thus,  in  normal  use  the  strip 
plays  multiple  roles,  including  its  role  of  legal  record 
with  its  mandated  marking  and  board  management 
responsibilities.  Compared  to  these  responsibilities, 
even  an  effective  memory  aid  would  be  difficult  to 
confirm.  The  Restricted  group,  freed  from  the  workload 
of  maintaining  the  strips,  may  have  been  able  to  focus 
in  a  more  optimal  fashion  on  the  most  relevant  infor¬ 
mation,  thus  more  than  compensating  for  the  lost 
memory  aids. 

Although  the  reduced-workload  hypothesis  is  clearly 
plausible,  it  is  unlikely  to  explain  the  prospective 
memory  effect  entirely  on  its  own.  For  example,  re¬ 
duced  workload  should  have  been  evidenced  by  the 
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Restricted  group  responding  to  the  secondary  probe 
more  quickly,  but  the  small  effect  is,  if  anything,  in  the 
opposite  direction. 

If  workload  alone  cannot  account  for  the  effect  of 
improved  prospective  memory,  what  may?  Why  in  the 
current  experiment  should  not  having  to  keep  a  legal 
record  manifest  its  benefits  in  prospective  memory? 
We  are  currently  working  on  the  hypothesis  that  the 
capacity  freed  by  eliminating  or  automating  board 
management  and  legal  maintenance  allow  the  control¬ 
ler  to  take  a  more  strategic  view  of  the  air  traffic.  This 
strategic  view  is  consistent  with  controllers  doing  bet¬ 
ter  in  planning  and  prospective  memory,  as  we  have 
shown  here,  but  it  also  implies  other  as  yet  untested 
hypotheses.  For  example,  a  controller  in  a  strategic,  as 
opposed  to  tactical  or  reactive  mode,  should  be  able  to 
see  conflictions  of  aircraft  further  into  the  future, 
should  be  better  able  to  identify  plausible  future  traffic 
configurations,  and  should  be  better  at  adjusting  to 
unexpected  traffic  situations.  The  degree  to  which  the 
static  paper  strip  can  serve  as  an  aid  for  these  processes 
may  not  be  much  beyond  that  supplied  by  watching 
the  data  block  move  in  real  time  on  the  PVD.  Unfor¬ 
tunately  it  is  difficult  to  reconcile  in  any  definitive  way 
the  discrepancy  between  our  finding  and  the  existing 
prospective  memory  literature  until  more  work  is  done 
on  prospective  memory  (e.g.,  expertise,  dynamic  tasks, 
analogical  depictions)  and  on  air  traffic  control  (e.g., 
cognitive  reallocation  of  resources,  effect  of  automat¬ 
ing  particular  subcomponents). 

Conclusion 

One  consequence  of  ISSS  will  be  a  reduction  in  the 
amount  of  direct  manual  interactions  with  the  flight 
progress  strips.  Overall,  it  seems  that  a  severe  reduc¬ 
tion  in  direct  interactions  with  flight  progress  strips 
will  not  have  dramatic  negative  consequences,  and 
may,  in  fact,  have  some  positive  ones:  Relieving  con¬ 
trollers  of  board  management  responsibilities  will  not 
adversely  affect  attention  and  retrospective  memory, 
and  will  have  a  positive  effect  on  at  least  some  manifes¬ 
tations  of  prospective  memory  and  perhaps  planning. 
The  current  results  also  suggest  that  research  and 
thinking  designed  to  anticipate  negative  effects  of 
automation  should  look  more  carefully  at  issues  relat¬ 
ing  to  the  integration  of  sub-tasks  within  the  overall 


flow  of  task  activities.  Although  this  research  bodes 
well  for  correctly-engineered  automation,  similar  stud¬ 
ies  should  use  controllers  other  than  Academy  instruc¬ 
tors,  should  use  scenarios  for  airspace  sectors  at  other 
en  route  centers,  and  should  attempt  to  compare 
specific  implementations  of  the  automation  (e.g.,  ISSS) 
when  it  becomes  available.  Until  then,  the  current 
research  suggests  that  less  interactions  with  flight 
progress  strips  should  not  be  viewed  as  a  large  handicap 
and  could  instead  enhance  the  controller’s  ability  to 
consider  future  events. 
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